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I.  INTRODUCTION 


The  U.S.  Department  of  Defense  (DOD)  Deputy  Assistant  Seeretary  of  the  Army  for  Research 
and  Technology  requested  that  the  Center  for  Technology  and  National  Security  Policy  (CTNSP) 
suggest  approaches  to  evaluating  the  quality  of  the  current  U.S.  Army  Science  and  Technology 
(S&T)  portfolio  and  the  quality  of  its  execution.  This  paper  contains  the  results  of  both  studies. 

There  are  two  aspects  of  the  U.S.  Army  S&T  program  that  are  evaluated  in  this  paper:  the 
portfolio  and  the  quality  of  its  execution.  This  paper  poses  the  questions  that  should  be  asked. 
Emphasis  for  the  analysis  of  the  portfolio  is  on  program  content,  the  distribution  of  resources 
over  the  component  parts  of  the  portfolio,  and  the  priorities  among  the  parts.  Normally  in  an 
assessment  of  this  type  it  is  necessary  to  identify  the  principal  stakeholders  and  customers  for  the 
research  and  development  program.  In  the  case  of  Army  S&T,  the  stakeholders  are  the  senior 
leaders  of  the  Army,  DOD,  Congress,  and,  ultimately,  the  American  taxpayer.  The  customers  are 
the  U.S.  Army  warfighters.  The  study  begins  by  outlining  a  clear  statement  of  the  vision  and 
mission  of  the  parent  organization — the  U.S.  Army.  We  continue  by  describing  in  detail  the 
needs  and  requirements  of  the  U.S.  Army  by  the  Training  and  Doctrine  Command  (TRADOC), 
the  Army  senior  leadership,  and  from  various  intermediary  customers  such  as  the  U.S.  Army 
Research,  Development  and  Engineering  Command  and  its  components,  and  the  Program 
Executive  Offices  and  the  Program  Managers  of  the  acquisition  community.  The  discussion  then 
turns  to  the  mechanisms  by  which  the  portfolio  is  assembled  and  approved,  as  well  as  the 
limitations  posed  by  the  realities  of  the  budget  details  of  resource  allocation. 

To  be  in  tune  with  the  Army’s  needs  now  and  into  the  future  means  that  the  portfolio  should 
have  a  balance  of  programs  between  long-term  basic  research,  medium-term  applied  research, 
and  short-term  work  that  can  be  transitioned  into  advanced  development  and  beyond.  The 
amount  of  work  in  each  category  will  vary  according  to  the  nature  of  the  individual  laboratory’s 
assignment.  Also  there  is  a  balance  between  the  work  done  in-house  and  that  done  extramurally 
on  contracts  and  grants  to  the  private  sector  or  by  other  government  laboratories. 

To  assess  the  quality  of  the  execution  of  the  program  requires  evaluating  the  strength  of  the 
Army  laboratory  system  in  terms  of  its  managers,  its  technical  staff,  and  available  facilities  and 
equipment.  This  evaluation  can  best  be  done  by  some  form  of  peer  review  by  subject  matter 
experts.  Additional  evaluations  should  be  made  by  the  Army  S&T  Office,  bringing  in  long-range 
technology  forecasting  to  assure  that  the  laboratory  system  will  be  prepared  for  future  demands. 
Some  form  of  benchmarking  is  needed  to  assess  where  the  best  work  can  be  done.  Collaborations 
can  then  be  established,  typically  by  contracts  and  grants  to  outside  laboratories. 

The  objective  is  to  have  the  strongest  possible  technology  program  appropriately  aligned  to  the 
needs  of  its  customers  and  the  expectations  of  its  stakeholders.  In  this  paper  we  present 
methodology  that  should  accomplish  this  goal.  In  chapter  II,  we  first  present  the  essential 
elements  of  the  Army  S&T  portfolio.  The  chapters  that  follow  describe  the  Army  S&T  portfolio 
and  the  evaluation  of  it,  the  evaluation  of  the  quality  of  the  program  of  execution,  and  a  summary 
of  the  report. 
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II.  THE  CURRENT  ARMY  S&T  PORTFOLIO 

Analysis  of  the  portfolio  should  begin  with  a  study  of  how  the  current  portfolio  came  into  being. 
It  is  based  on  a  long  series  of  budget  decisions  made  by  the  Army  and  Congress.  These  decisions 
were  based  on  the  threats  facing  the  Army  at  the  time  and  with  forecasts  of  likely  threats  in  the 
future.  When  we  are  at  war,  the  laboratories  are  necessarily  more  focused  on  the  short  term.  In 
the  2000s  the  intense  focus  on  the  wars  in  Afghanistan  and  Iraq  and  development  of  the  Future 
Combat  System  shortened  the  time  frame  of  the  S&T  program.  Recent  studies  have  expressed 
concern  that  the  current  time  frame  is  too  short.  ^  The  Army  is  now  undergoing  a  period  of  force 
rebalancing  and  downsizing.  The  mission  is  shifting  away  from  counterinsurgency  in 
Afghanistan  and  Iraq  to  the  Asia-Pacific  region. 

With  the  end  of  the  Cold  War,  the  Army  underwent  a  dramatic  downsizing.  The  S&T  program 
was  markedly  reduced,  sometimes  by  as  much  as  one-half  in  budgetary  terms.  The  reduction  in 
the  laboratory  budgets  removed  many  programs  of  lower  priority;  overhead  costs  were  reduced 
to  a  minimum.  This  belt  tightening  left  the  most  important  research  programs  intact.  The  past 
decade  of  counterinsurgency  led  to  some  of  the  budget  reductions  being  restored  to  mount 
programs  directed  at  the  wars  in  the  Middle  East. 

Also  during  the  1990s  and  beyond,  the  Army  S&T  management  turned  outward  in  the  sense  that 
it  established  some  very  large  extramural  collaborations.  Centers  of  excellence  included  the 
Institute  for  Soldier  Nanotechnology,  the  Institute  for  Creative  Technologies,  and  the  Institute  for 
Collaborative  Biotechnologies.  The  Army  also  has  funded  the  Institute  for  Applied  Technology 
at  the  University  of  Texas  for  studies  of  electric  guns.  The  focus  of  this  work  has  recently  been 
transferred  to  the  United  States  navy.  These  institutes  are  university-based,  involve  several 
different  disciplines,  and  consist  of  formal  or  informal  consortia  from  industry  and  academe.  A 
similar  approach  was  used  in  the  creation  of  several  extramural  consortia  comprising  academic 
and  industrial  members  at  the  Army  Research  Laboratory  (ARE).  These  consortia  were  tied 
closely  to  in-house  ARE  programs  in  that  normal  grants  and  contracts  are  operated  at  arm’s 
length  from  the  in-house  staff,  that  is,  once  the  contract  or  grant  is  awarded,  interaction  with 
Army  personnel  is  not  required  until  renewal  discussions  are  held.  These  ARE  collaborative 
centers  of  excellence — Collaborative  Technology  Alliances  and  Collaborative  Research 
Alliances — differ  from  ordinary  external  centers  of  excellence  in  that  they  are  designed  to 
operate  parallel  with  in-house  research  conducted  by  Army  laboratory  staff  with  the  very  best 
people  in  the  private  sector.  This  type  of  management  involves  rotation  of  staff  into  and  out  of 
the  Army  laboratory;  such  rotation  both  improves  mutual  understanding  of  the  requirements  for 
research  and  speeds  transfer  of  new  technical  information.  These  collaborations  are  intended  to 
ensure  that  the  Army  knows  and  can  work  with  the  best  and  the  brightest  people  elsewhere.  The 
result  is  that  the  work  is  first  class,  and  the  Army  technical  staffs  are  able  to  adapt  and  help  adopt 
these  new  developments  for  use  in  other  sectors  of  the  Army. 


*  Report  of  an  Expert  Panel  on  the  Future  of  Army  Laboratories,  Improving  Army  Basic  Research  (Arlington,  VA: 
RAND,  2012);  Final  Report  of  the  Army  2010  Army  Acquisition  Review  Chartered  by  the  Secretary  of  the  Army, 
Army  Strong:  Equipped,  Trained  and  Ready  (Washington,  DC:  Office  of  the  Secretary  of  the  Army,  2011). 
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The  current  Army  S&T  portfolio  has  seven  segments: 

•  Basic  Research 

•  The  Soldier  (nonmedical) 

.  The  Soldier  (medical) 

.  Ground 

.  Command,  Control,  Communications,  and  Intelligence  (C3I) 

.  Air 

.  Enduring  Technologies. 

Each  of  these  segments  is  subdivided  into  six  or  seven  component  programs.  Eor  this  discussion, 
we  classify  the  research  in  three  categories:  Army  Unique,  Army  Shared,  and  Dominated  by 
Others.  Army  Unique  covers  work  that  others  have  little  or  no  motivation  to  work  on  and  the 
subject  matter  is  critical  for  the  Service.  In  these  circumstances,  the  Army  maintains  a  robust 
research  program  in  house  benefiting  occasionally  from  work  done  elsewhere.  There  will 
normally  be  a  strong  basic  research  component  to  ensure  that  the  Army  anticipates  and  is 
prepared  for  future  developments  in  the  technology  and  in  threats  involving  that  technology. 

Army  Shared  includes  research  topics  where  both  the  Army  and  others  are  motivated  to  push  the 
state  of  the  art.  Such  topics  include  sensors,  hardened  electronics,  and  situational  awareness  on 
the  battlefield.  Eor  these  types  of  technologies,  the  Army  has  active  research  programs 
addressing  specific  applications  important  to  the  Army  and  where  the  work  of  others  is  not 
focused  on  Army  interests.  The  Army  work  is  tailored  to  meet  specific  requirements  provided  by 
the  representatives  of  the  warfighters.  Eong-term  basic  research  is  limited  to  these  specific  needs 
and  to  solving  unanticipated  problems  that  arise  in  the  Research  and  Development  program.  In 
shared  areas,  various  forms  of  collaboration  are  employed,  ranging  from  informal  cooperation  on 
specific  points  to  formal  collaborations  under  contract  agreements.  The  Collaborative 
Technology  Alliances  and  the  Collaborative  Research  Alliances  fall  in  the  latter  category. 

The  third  category.  Dominated  by  Others,  includes  technology  where  the  private  sector  or 
another  part  of  the  government  is  dominant  and  the  Army  relies  on  them  for  research.  Eor 
example,  in  the  development  of  new  computers,  the  Army’s  role  in  research  is  to  help  the 
acquisition  community  adapt  the  technology  to  special  needs  of  the  warfighter.  It  is  unlikely  that 
there  will  be  a  long-term  basic  research  program  in  this  category. 

Parsing  the  portfolio  into  these  categories,  however,  is  judgmental;  people  will  differ  about 
where  to  place  a  particular  portfolio  element.  Eor  example,  the  desire  of  the  Soldier  to  see  well  at 
night  led  to  the  Army  developing  a  dominant  position  in  night  vision  technology.  Army  success 
produced  market  opportunities  in  the  commercial  sector  so  that,  today,  there  is  technical 
competence  in  industry  as  well.  It  is  also  true  that,  in  fielding  a  technology  such  as  night  vision, 
industry  has  to  learn  how  to  manufacture  the  devices  and  components  the  Army  needs.  So  there 
is  expertise  in  various  companies  fostered  by  the  Army.  But  the  Army  is  maintaining  its 
technical  lead  in  night  vision  by  developing  and  evaluating  new  concepts  for  next  generations. 
Thus  today  both  the  Army  and  industry  are  competent  in  night  vision  technology,  even  though 
the  Army  will  continue  to  press  new  developments. 
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Another  example  is  in  armor  for  the  main  battle  tank.  The  armor  for  the  Abrams  tank  grew  out  of 
some  work  done  in  the  United  Kingdom,  the  Chobham  armor  concept.^  The  Army  was  working 
on  this  in  parallel  with  the  British.  Subsequently  Army  researeh  added  more  modifieations  so 
that  today  Army  armor  leads  the  teehnology.  Given  the  importanee  of  survivability  of  the  tank 
and  its  erew  the  Army  is  maintaining  its  research  on  armor  and  is  pushing  the  limits  wherever 
and  however  it  can. 

The  situation  with  armaments  is  similar.  The  Army  has  always  had  technical  programs  to 
improve  its  weapons  and  holds  the  lead  for  many.  The  work  at  Benet  Laboratory  and  Watervliet 
Arsenal  has  given  the  Army  a  strong  position  in  developing  and  manufacturing  large  bore  guns 
both  for  itself  and  for  others  in  the  military.  There  is  no  motivation  for  the  private  sector  to 
develop  cannon  technology.  Current  problems  being  addressed  are  in  manufacturing  materials, 
material  coatings,  and  in  manufacturing  processes.  An  example  of  the  importance  of  solving 
these  problems  is  that  the  dimensional  precision  and  accuracy  of  a  gun  tube,  along  with  its 
stability  under  sustained  firing,  contributes  to  the  accuracy  of  the  weapon.  Research  on  the 
factors  affecting  these  errors  have  been  a  major  focus  over  the  years,  and  are  a  special  niche  for 
the  Army  weapons  research  program.  One  solution  to  this  problem  was  the  development  of  GPS 
fuzes  for  artillery  rounds  to  reduce  the  number  of  rounds  needed  to  hit  a  target.  However,  it  took 
time  to  create  a  fully  functioning  weapons  system.  Early  models  only  showed  the  path  of  the 
round  and  where  it  would  hit,  while  later  efforts  added  guidance  to  the  fuze  ensuring  a  very  large 
reduction  in  the  impact  error  calculations.  The  result  is  the  Excalibur  artillery  round,  which  has 
been  fielded  in  Afghanistan.  The  design  of  the  fuze  is  such  that  it  can  be  screwed  on  to  rounds  in 
the  current  stockpile.  This  is  an  example  of  an  “Army  Unique”  capability. 


^  Richard  Chait,  John  W.  Lyons,  and  Duncan  Long,  Critical  Technology  Events  in  the  Development  of  the  Abrams 
Tank:  Project  Hindsight  Revisited,  Defense  &  Technology  Paper  22  (Washington,  DC:  Center  for  Technology  and 
National  Security  Policy,  December,  2005). 
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III.  THE  BASICS  OF  EVALUATING  THE  PORTFOLIO 


The  process  of  evaluating  the  portfolio  may  be  done  by  internal  staff  at  the  office  of  the  Deputy 
Assistant  Secretary  of  the  Army  for  Research  and  Technology,  by  others  in  the  Army  Secretariat, 
by  the  Office  of  the  Secretary  of  Defense,  or  by  external  experts.  In  any  case,  the  review  should 
proceed  somewhat  as  described  in  the  following  sections  of  this  paper.  A  first  step  is  to  ask  a 
series  of  questions  such  as  the  following: 

•  How  does  the  work  support  the  Army  mission? 

.  What  is  the  priority  given  to  the  area  by  TRADOC  and  Army  leaders? 

.  Is  the  proposed  technology  essential  or  just  nice-to-have? 

.  What  is  the  projected  impact?  Compared  to  other  Army  S&T  programs? 

.  What  is  the  relation  to  forecasted  trends  in  technology  and  in  threats? 

.  Is  the  area  within  the  scope  of  the  Army’s  current  programs?  Is  it  within  the  expertise  of 
the  S&T  staff? 

.  Can  the  Army  develop  collaborations  with  others  already  competent  in  the  field? 

.  Can  the  private  sector  or  other  government  entities  provide  the  technology  needed?  If  so, 
what  and  how  much  research  would  be  needed  to  adapt  their  work  to  Army  needs? 

.  What  would  the  absence  of  Army  in-house  expertise  mean  to  the  warfighter? 

.  What  is  the  Army’s  expertise  in  this  area? 

.  What  expertise  exists  elsewhere  in  the  military  or  the  private  sector? 

.  Is  the  Army  dominant  in  the  field? 

.  What  would  happen  if  the  Army  were  to  stop  research  in  this  field?  Are  other  sources 
ready  and  able  to  do  the  work?  At  what  cost  in  time  and  money? 

.  Is  the  size  of  the  Army  program  at  a  critical  mass? 

•  Is  the  budget  for  personnel,  facilities,  and  capital  equipment  adequate  for  the  job? 

.  Are  the  necessary  resources  already  on  board  (for  a  new  project  launch)? 

.  Is  there  provision  for  basic  research  to  support  the  programs  and  ensure  it  is  at  the 
frontiers? 

•  How  competent  are  Army  personnel  in  this  field  as  measured  by  publications,  patents, 
recognition,  etc.  compared  to  other  similar  facilities? 

•  How  will  progress  be  assessed?  Are  there  criteria  for  continuing  or  termination? 

To  elaborate,  consider  the  following  approach.  The  review  package  and  briefings  should  have 
the  Component  Technical  Areas  and  Technical  Descriptors.  The  Descriptors  would  include 
technical  products  that  fill  desired  capabilities,  list  the  Army  contributors  and  funding,  and  the 
stakeholders.  The  final  section  should  address  four  major  categories:  the  quality  of  the  technical 
staff  and  the  technical  managers,  the  relevance  of  the  program,  alignment  of  the  program  with 
the  Army  mission  and  priorities,  and  consistency  with  Army  forecasts  of  future  developments  in 
technology. 

As  an  example,  consider  the  area  of  C3I  (Command,  Control,  Communications,  and 
Intelligence),  a  technical  area  shared  with  industry. 

.  Technical  areas:  intelligence  and  electronic  sensors,  other  sensors,  communications,  and 
mission  command 

.  Products:  sensors  for:  active/passive  UV,  visible,  and  Electro-Optical/Infra-Red; 
terahertz,  mmw,  radar,  acoustic,  magnetic,  seismic,  olfactory,  and  tactile  sensors;  sensor 
components,  multisensors,  sensor  processing,  and  data  fusion 
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•  Capabilities;  reconnaissance,  intelligence,  surveillance,  and  target  acquisition  (RISTA), 
navigation,  pilotage,  persistence  surveillance,  forensics,  human  intent  and  identification, 
health  monitoring,  early  warning,  missile  seekers,  and  munitions  guidance,  detection  of 
improvised  explosive  devices,  countermine,  chem./bio  hazards,  and  robot  perception 
under  all  weather  conditions  and  terrains,  and  through  walls. 

Contributors 

1 .  Communications  and  Electronics  Research,  Development,  and  Engineering  Center:  Night 
Vision  and  Electronic  Sensors  Directorate,  Intelligence  and  Electronic  Warfare 
Directorate 

2.  Army  Research  Eaboratory:  Sensors  and  Electron  Devices  Directorate,  Communications 
and  Information  Sciences  Directorate,  Human  Research  and  Engineering  Directorate 

3.  Tank  Automotive  Research,  Development,  and  Engineering  Center 

4.  Armament  Research,  Development,  and  Engineering  Center 

5.  Aviation  and  Missile  Research,  Development,  and  Engineering  Center 

6.  Edgewood  Chemical  and  Biological  Center 

7.  Natick  Soldier  Research,  Development,  and  Engineering  Center 

8.  Corps  of  Engineering  Research  and  Development  Center 
Customers 

1 .  Program  Executive  Office  (PEO)  Aviation 

2.  PEO  Ground  Combat  Systems 

3.  PEO  Missiles  and  Space 

4.  PEO  Soldier 

5.  PEO  Chemical/Biological 

6.  PEO  Command  Control  Communications-Tactical 

7.  Medical  Research  and  Materiel  Command 

8.  Training  and  Doctrine  Command 

9.  Eogistics  Innovation  Agency 

10.  Defense  Threat  Reduction  Agency 

1 1 .  Edgewood  Chemical  and  Biological  Center 

12.  Special  Operations  Command 

13.  Joint  Improvised  Explosive  Devices  Defeat  Organization 

14.  Rapid  Equipping  Eorce 

15.  Department  of  Homeland  Security 

16.  Department  of  Energy. 
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The  data  needed  for  evaluating  the  effort  would  be  provided  by  the  eontributors  and  eustomers 
listed  above.  ^  The  questions  to  be  answered  are,  for  example,  those  at  the  beginning  of  this 
ehapter. 

Some  Preliminary  Assessments 

Ground  and  Air.  The  contents  of  the  Ground  portfolio  suggest  that  most  of  those  items  are  Army 
unique.  Weapons,  survivability,  dealing  with  countermine/  improvised  explosive  devices  and 
technical  improvements  to  platforms  are  necessary  for  the  warfighter.  More  effective  sources  of 
power,  unmanned  systems,  and  logistics  are  areas  where  the  private  sector  has  much  to  offer. 
Some,  such  as  alternate  fuels,  are  being  addressed  in  very  large  programs  by  the  U.S. 
Department  of  Energy.  Subsequently,  the  Army  should  not  invest  heavily  in  this  area  except  to 
adapt  the  findings  to  its  needs.  For  the  Air  portfolio,  the  design  and  structure  of  helicopters  is  a 
shared  area  with  industry.  This  is  recognized  in  the  Army’s  National  Rotorcraft  Technology 
Center  where  cooperation  among  the  players  is  the  norm.  Most  of  the  topics  are  of  special 
interest  to  the  Army;  some  are  also  of  interest  to  the  private  sector.  Examples  of  the  latter  are 
better  fuel  efficiency  for  the  helicopter  engines,  lightweight  drive  trains,  improved  reliability  and 
durability,  and  reduced  weight  and  vibration.  Unmanned  aerial  vehicles  are  a  topic  of  the  highest 
priority  for  the  Air  Force  at  the  current  time,  and  the  Army  will  learn  from  their  work  and  adapt 
it  to  its  fleet. 

C3I.  Sensors,  communications/computers,  and  command  and  control  are  essential  items  for  the 
battlefield.  Some  will  be  specific  to  the  Army,  while  others  are  of  interest  to  the  other  services 
and  to  our  military  partners  around  the  world.  Everyone  in  and  out  of  the  military  is  interested  in 
sensors.  New  developments  in  solid  state  physics  continue  to  be  made.  The  technology  is  moving 
to  nano-scale  devices.  All  of  these  influences  will  have  an  effect  on  future  technology.  Seen 
together,  the  impact  on  night  vision,  detectors  for  sensors  on  platforms  of  all  types,  and  force 
protection  warning  devices  will  help  improve  the  detection  of  movements,  the  launch  of  mortar 
shells,  and  sniper  bullets  on  the  battlefield.  Communications  and  computer  technology  derive 
from  the  work  of  that  industry. 

Developments  in  computers  are  the  province  of  industry,  but  software  is  another  matter.  Special 
programs  are  needed  as  real-time  decision  aids  for  battlefield  commanders.  Many  technologies 
are  needed  for  the  gathering  and  analysis  of  intelligence  information.  The  amount  of  data 
collected  by  today’s  satellites  and  surveillance  aircraft  is  creating  a  need  for  better  methods  of 
mining  and  assessing  this  information  in  real-  or  near-real  time.  This  is  essential  for  today’s 
Army  and  will  be  even  more  so  in  the  near  future.  However,  data  mining  is  of  great  interest  in 
other  agencies  and  the  private  sector.  Researchers  in  astronomy  and  in  atmospheric  science  deal 
with  very  large  amounts  of  data  coming  in  everyday,  sometimes  every  hour.  Thus,  research  in 
this  area  should  be  left  largely  to  those  sectors.  Techniques  for  the  fusion  of  data  from  different 
sources  are  also  needed.  A  need  highlighted  in  Iraq  is  for  devices  that  can  reveal  the  contents  of 
buildings  either  by  seeing  through  the  walls  or  by  entering  surreptitiously  via  micro-autonomous 
robots.  Research  in  both  approaches  is  active  in  the  Army. 

The  Soldier.  This  portfolio  is  separated  into  two  parts:  one  for  nonmedical,  the  other  for  medical. 
The  nonmedical  section  is  divided  into  two  subsections:  the  integration  of  humans  with  machines 
and  measures  of  effectiveness  regarding  that  integration,  and  how  to  improve  the  design  of 


^  For  possible  metrics,  see  Report  of  an  Expert  Panel  on  the  Future  of  Army  Laboratories,  Improving  Army  Basic 
Research  (Arlington,  VA:  RAND,  2012),  50-51. 
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equipment.  Both  these  topics  are  of  interest  to  many  in  and  out  of  government  because  it  is  of 
pressing  importance  to  soldiers  on  the  battlefield  and  therefore  there  is  justification  for  some 
work  in  the  Army.  Another  area  that  has  arisen  recently  is  that  of  small  unit  operations.  The 
Army  is  moving  to  a  focus  on  independence  in  units  as  small  as  the  squad.  Studies  are  currently 
underway  on  communications  within  small  units,  as  well  as  upward  to  high  echelons.  This  has 
been  and  is  especially  important  in  operations  in  dense  urban  areas.  The  selection  of  Army 
warfighters  and  the  training  necessary  to  make  them  effective  is  always  a  priority.  This  includes 
finding  ways  to  determine  the  toughness  and  resilience  of  new  personnel  to  help  improve  their 
ability  to  surmount  post-traumatic  stress  disorder  and  to  maintain  a  high  level  of  effectiveness  on 
the  battlefield.  As  this  is  far  more  prevalent  in  ground  soldiers  than  members  of  other  services,  it 
may  be  of  more  importance  to  the  Army  than  to  other  services. 

Medically,  survivability  of  the  troops  and  care  of  combat  casualties  are  perhaps  the  top  priorities 
of  Army  commanders.  Topics  such  as  improved  body  armor  for  protection  of  the  head  to 
minimize  traumatic  brain  injuries  are  important  priorities  for  the  Army  Medical  Command.  Part 
of  this  is  improving  the  resistance  of  soldiers  to  infectious  diseases.  On  the  battlefield,  ways  are 
needed  to  monitor  the  vital  signs  of  soldiers  as  they  work  and  after  they  are  injured.  Research  is 
ongoing  on  fabrics  for  uniforms  that  perform  monitoring  while  serving  the  usual  functions  of 
cloth.  Better  means  of  stemming  hemorrhage  have  been  developed  during  the  Middle  East 
conflicts.  This  is  a  top  priority  in  treating  wounded  on  the  battlefield.  Finding  markers  in  blood 
that  can  signal  internal  damage  in  the  brain  or  elsewhere  is  a  dream  of  medical  researchers. 
Currently  sufferers  from  traumatic  brain  injuries  may  be  undiagnosed  until  much  later.  The 
Army  seeks  a  biological  marker  such  that  a  simple  blood  test  would  reveal  brain  damage  long 
before  there  would  be  visible  symptoms;  this  would  enable  early  diagnosis  and  treatment.  It 
should  be  noted  that  there  is,  in  the  medical  research  program,  a  large  amount  of  research  that  is 
not  directly  connected  to  the  needs  of  the  warfighter.  A  notable  example  is  the  DOD  research  on 
breast  cancer,  a  program  managed  by  the  Army  Medical  Command’s  Medical  Research  and 
Materiel  Command  (MRMC).  Finally,  the  Army  is  funding  academic  consortia  to  develop  organ 
regeneration  science  and  technology.  The  ultimate  aim  is  to  help  soldiers  suffering  amputations. 
A  related  technology  is  the  control  of  prostheses  by  electronic  connections  to  the  nervous 
systems  at  appropriate  levels.  Some  success  has  already  been  achieved  with  the  Defense 
Advanced  Research  Projects  Agency  leading  the  way.  The  Army  medical  research  program  is 
closely  linked  to  non-military  medical  research,  including  pharmaceutical,  and  depends  on  the 
community  for  various  efforts  required  to  achieve  Food  and  Drug  Administration  approvals. 
Collaboration  and  cooperation  is  the  rule. 

Enduring  Technologies  Portfolio.  The  work  on  environmental  quality  at  installations,  including 
clean-up  on  ranges  and  at  former  dumps,  is  important  for  compliance  with  environmental  rules 
and  regulation.  However  Army  research  is  relatively  minor.  Pollution  is  an  issue  that  society  in 
general  shares.  Disposal  of  chemical  weapons  has  been  and  still  is  of  major  concern.  Much  of  the 
research  done  in  this  area  is  engineering  work,  with  analytical  methods  the  subject  of  some 
study.  These  analytical  methods  are  needed  to  ensure  that  not  even  the  least  amount  of  toxins  is 
released  during  disposal. 

Much  of  the  Army’s  physics-based  modeling  and  simulation  work  is  done  on  high  performance 
computers.  A  principle  source  of  such  computing  is  provided  by  the  High  Performance 
Computing  Modernization  Program,  now  managed  by  the  Corps  of  Engineers  at  Vicksburg. 
They  manage  centers  of  excellence  across  DOD.  This  does  not  involve  research  on  developing 


better  supercomputers;  that  is  left  to  industry  and  academe.  The  centers  provide  the  means  to 
work  on  problems  that  are  intractable  on  less  capable  computers.  There  is  also  very  good  work 
on  network  science  for  the  battlefield  that  relates  to  work  in  the  private  sector  and  in  academe. 

Basic  Research.  We  consider  here  three  categories  of  basic  research:  in-house  research,  and  two 
types  of  extramural  research.  The  in-house  research,  conducted  by  Army  technologists,  may  be 
long-term  exploratory  work  pushing  the  frontiers,  or  relatively  short-term  study  to  elucidate 
phenomena  raising  barriers  to  progress  on  current  projects.  In  neither  case  does  the  DOD 
definition  of  6.1  research  apply.  Namely,  the  work  is  curiosity  work  with  no  specific  application 
in  mind.  The  two  extramural  categories  include  single-investigator  grants,  typically  to 
universities,  and  sponsorship  of  large  centers  of  excellence.  These  latter  include,  under  the  rubric 
of  University  Affiliated  Research  Consortia,  two  sets  of  programs  called  Collaborative 
Technology  Alliances  and  Collaborative  Research  Alliances  established  by  ARL  and  designed  to 
bring  together  the  external  consortia  members  with  in-house  Army  technical  staff.  (External 
centers  of  excellence  are  not  so  closely  related  to  Army  laboratories.) 

ARL  is  the  leader  in  Army  basic  research  both  in  in-house  work  and,  since  the  Army  Research 
Office  (ARO)  is  part  of  ARL,  in  extramural  research.  In  recent  years  the  amount  of  in-house 
basic  research  at  ARL  has  been  reduced  by  transferring  6.1  funds  to  the  Collaborative 
Technology  Alliances. 

A  lengthy  study  of  basic  research  in  the  Army  by  the  Office  of  the  Deputy  Assistant  Secretary  of 
the  Army  for  Research  and  Technology  is  currently  underway  looking  at  the  portfolio.  In 
particular,  the  study  is  looking  at  the  selection  of  basic  research  areas  and  the  scenarios  and 
technology  gaps  derived  from  the  analyses  of  TRADOC.  This  study  is  focused  on  the  grants 
made  by  ARO  and  will  lead  to  a  more  formal  articulation  of  the  Office’s  philosophy  and 
management  approach.  This  study  and  a  companion  study  of  the  formulation  of  the  Army’s 
applied  research  program  will  produce  some  revamping  of  the  procedures  and  programs  to  make 
them  more  understandable  to  the  warfighters  and  the  Army’s  senior  leadership."^ 

The  Basic  Research  Portfolio.  The  basic  research  portfolio  in  the  current  analysis  is  in  four 
component  parts:  platform  centric;  material  centric;  information  centric;  and  human  centric. 

ARO  sponsors  research  that  is  intended  to  push  the  frontiers  of  already  state-of-the-art  science 
and  technologies,  and  to  identify  and  pursue  new  areas  of  research  that  may  be  useful  to  the 
Army  in  the  future.  Projects  in  the  sciences  include  study  of  new  concepts  in  physics,  chemical 
sciences,  and  materials,  each  covering  a  number  of  subareas  that  are  at  the  frontiers  of 
knowledge.  The  current  summary  of  the  portfolio  supplied  by  the  Office  of  the  Deputy  Assistant 
Secretary  of  the  Army  for  Research  and  Technology  is  arranged  by  major  topics,  not  by 
organization.  Thus  there  is  no  mention  of  ARO’s  basic  research  in  biotechnology,  neuroscience, 
and  other  areas  in  life  science  now  being  pressed  in  the  scientific  community  at  large.  A  more 
detailed  study  would  analyze  these  areas  as  well. 


^  Dr.  Marilyn  Freeman,  The  United  States  Army ’s  Science  and  Technology  (S&T)  Program  for  Fiscal  Year  2013, 
Statement  to  the  U.S.  Flouse  of  Representatives,  Committee  on  Armed  Services,  Subcommittee  on  Emerging 
Threats  and  Capabilities,  Second  Session,  1 12*  Congress,  February  29,  2012,  available  at 
<http://defenseinnovationmarketplace.miFresources/Freeman_Testimony_2013.pdf  >. 
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ARO  specifically  excludes  certain  sub-branches  of  areas,  although  it  does  ensure  some  room 
remains  available  for  unusual  opportunities  that  could  be  useful  for  the  Army: 

The  Physics  Division  has  little  direct  interest  in  relativity  and  gravitation,  cosmology, 
elementary  particles,  nuclear  physics,  astronomy,  or  astrophysics,  since  they  generally  have 
little  impact  on  the  areas  of  Army  needs.  Nevertheless,  the  possible  relevance  of  topics 
within  these  other  physics  disciplines  is  not  absolutely  discounted,  and  discussions  of 
potential  exceptions  are  welcome.^ 

An  inspection  of  the  science  topics  ARO  is  interested  in  reveals  a  broad  range  of  research  areas, 
each  of  which  is  related  to  some  aspect  of  the  Army’s  mission.  ARO’s  Physics  Division  states: 

Condensed  Matter  Physics  (CMP)  is  a  foundational  science  enabling  fundamental  Army 
technologies  in  areas  such  as  information  processing,  communications,  sensors,  optical 
components,  electronics,  optoelectronics,  night  vision,  seekers,  countermeasures,  and  many 
others.  Technologies  such  as  these  would  not  exist  today,  at  least  not  as  we  know  them, 
without  visionary  research  in  the  field  of  CMP.  The  ARO  CMP  program  strives  to  continue 
this  level  of  impact  by  looking  beyond  the  current  understanding  of  natural  and  designed 
condensed  matter,  to  lay  a  foundation  for  revolutionary  technology  development  for  next 
generation  and  future  generations  of  warfighters.^ 

The  MRMC  performs  or  sponsors  basic  research  in  many  different  areas.  Recently  it  has 
established,  with  funding  support  from  a  number  of  other  federal  agencies,  two  research 
consortia  in  a  number  of  universities  on  tissue  and  organ  regeneration.  Each  of  these  civilian 
consortia  is  itself  a  multi-institutional  network.^  MRMC  conducts  programs  on  traumatic  brain 
injury,  blood  loss  control,  pain  management,  and  control  of  and  immunization  against  a  variety 
of  infectious  diseases  that  are  hazards  to  soldiers.  These  last  named  programs  are  generally 
carried  out  cooperatively  with  the  pharmaceutical  industry  and  the  Food  and  Drug 
Administration.  MRMC  has  a  long  and  notable  history  in  developing  and  demonstrating 
techniques  for  combat  casualty  care  on  the  battlefield  and  for  subsequent  long-term  medical  care 
and  rehabilitation  for  wounded  and  traumatized  soldiers. 

The  Corps  of  Engineers  Engineering  Research  and  Development  Center  has  a  number  of  basic 
research  programs — many  in  environmental  areas  such  as  remote  sensing  of  terrain — and  it 
conducts  modeling  and  simulation  studies  on  its  High  Performance  Computing  Shared  Resource 
Facility.  The  Engineering  Research  and  Development  Center  also  manages  DOD’s  High 
Performance  Computing  Modernization  Office.  A  group  within  this  Office  manages  a  program 
called  CREATE  that  is  “developing  and  deploying  scalable,  multiphysics-based  computational 


^  Army  Research  Laboratory,  Army  Research  Laboratory  Broad  Agency  Announcement  for  Basic  and  Applied 
Scientific  Research,  W91 1NF-12-R-001 1,  15  May  2012-31  March  2017,  93,  available  at 
<www.arl.army.mil/www/pages/8/research/Final_Post_ARL_BAA_W911NF-12-R-0011.pdf>. 

'’Ibid.,  93. 

’  Armed  Forces  Institute  of  Regenerative  Medicine,  homepage,  available  at 

<https://mrmc.amedd.army.miFindex.cfm?pageid=medical_r_and_d.afirm.overview>.  One  consortia  is  led  by  the 
Wake  Forest  University  Baptist  Medical  Center  along  with  the  McGowan  Institute  for  Regenerative  Medicine  at 
Pittsburgh;  the  other  is  led  by  Rutgers  University,  the  State  University  of  New  Jersey,  along  with  the  Cleveland 
Clinic. 
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engineering  tools  to  design  and  analyze  DOD  weapons  system  performance.”  This  is 
exploratory  work  pushing  the  frontier  in  a  new  area  for  DOD  computing,  namely,  becoming 
involved  directly  in  the  engineering  and  manufacturing  end  of  the  acquisition  timeline.  The  use 
of  high  performance  computing  in  this  way  in  the  private  sector  is  shown  to  shorten  the  time  to 
complete  new  product  development  by  as  much  as  a  factor  of  three.  ^  This  result  is  made  possible 
by  taking  advantage  of  the  power  of  these  machines  to  model  complex  systems  that  have  been 
beyond  the  reach  of  less  powerful  computers. 

Portfolio  Summary.  The  S&T  portfolio  is  the  result  of  many  studies  and  decisions  over  many 
years.  It  has  been  revised  and  shaped  by  the  changing  needs  of  the  warfighter,  as  well  as 
continuing  discussions  between  the  S&T  community  and  the  warfighters  as  represented  by 
TRADOC  at  policy  levels.  It  also  maintains  contacts  between  members  of  the  Army  laboratory 
system  and  soldiers  at  home  and  abroad.  TRADOC  regularly  issues  the  top  Army  challenges. 
The  S&T  program  planners  update  their  priorities  accordingly  and  add  new  programs  (and 
terminate  older,  lower  priority  work)  to  maintain  alignment  of  the  S&T  programs  with  doctrinal 
priorities.  For  short-term  work,  the  alignment  must  be  very  close.  Programs  in  advanced 
development  are  tailored  to  respond  to  a  list  of  24  Army  S&T  Challenges.  Somewhat  longer- 
range  work  in  applied  research  anticipates  the  evolution  of  the  problems.  Long-term,  basic 
research  (6.1)  seeks  to  uncover  new  opportunities  at  the  frontiers  of  scientific  and  engineering 
knowledge.  The  Army  has  established  centers  of  excellence  in  basic  research  and  early  applied 
work  based  on  present  needs,  but  with  the  flexibility  to  pursue  new  leads. 

The  following  is  a  list  of  areas  which,  in  our  judgment,  occupy  a  special  niche  in  Army 
technology  and  therefore  should  receive  priority  support: 

•  Soldier  and  vehicle  armor  and  armaments 

.  Night  vision  equipment 

.  Unattended  ground  sensors  and  sensor  networks 

.  Air  acoustic,  seismic,  and  magnetic  sensors 

.  Terrain  and  topographical  analysis  and  navigation 

•  Soldier  power  and  thermal  management  of  platforms 

.  Helicopter  protection  and  lethality 

•  Individual  soldier  health  sensing  and  monitoring 

•  Individual  soldier  robotic  associates 

.  Mobile  ground  communications  networks 

.  Mine/IED  detection  and  clearing  at  operational  tempo 

•  Tactical  chemical/biological/nuclear  detection  and  protection 

•  Ground  combat  survivability/vulnerability  analysis. 


*  John  W.  Lyons,  Richard  Chait,  Charles  J.  Nietubicz,  The  Use  of  High  Performance  Computing  (HPC)  to 
Strengthen  the  Development  of  Army  Systems,  Defense  &  Technology  Paper  87  (Washington,  DC:  Center  for 
Technology  and  National  Security  Policy,  November  2011). 

^  Loren  Miller,  quoted  in  Council  on  Competitiveness,  Goodyear  Puts  the  Rubber  on  the  Road  with  High 
Performance  Computing,  Case  study,  2009, 4,  Available  at 

<www.compete.org/images/uploads/File/PDF%20Files/HPC_Goodyear_032009.pdf>. 

***  See  Army  Science  &  Technology  problems  and  Challenges,  PowerPoint,  undated,  available  at 
<http://usarmy.vo.llnwd.net/e2/c/downloads/239849.pdf>. 
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IV.  EVALUATING  THE  QUALITY  OF  THE  PROGRAM 

AND  ITS  EXECUTION 


The  second  part  of  the  Army  S&T  program  review  by  the  Deputy  Assistant  Secretary  of  the 
Army  for  Research  and  Technology  is  seeking  answers  to  the  following  questions: 

•  How  does  one  assess  the  execution  of  the  program  with  existing  research  staff  and 
infrastructure? 

•  Do  we  have  the  best  personnel  and  infrastructure  compared  to  leading  research  and 
development  programs? 

In  this  section  of  the  paper  we  develop  a  methodology  for  making  these  evaluations.  We  begin 
with  a  discussion  of  the  factors  that  are  necessary  to  achieve  excellence  in  a  research  laboratory. 
Next  is  a  review  of  the  practice  of  expert  review  of  the  laboratory  and  its  work  by  unbiased 
subject  matter  experts.  We  then  consider  the  relevance  of  the  work  and  the  value  of  continued 
contact  with  customer  and  ultimate  users  of  the  results  from  the  laboratories.  We  consider  the 
need  for  and  value  of  formal  and  informal  collaborations  with  other  laboratories.  All  of  this 
discussion  leads  to  the  role  of  excellence  in  a  laboratory  to  its  standing  in  the  community  and  to 
the  support  the  laboratory  has  in  the  senior  leadership  in  the  Army  and  the  authorizing  and 
appropriating  committees  in  the  Congress. 

The  quality  of  a  program  of  research  and  development  depends  on  many  factors,  but  the  most 
significant  are  the  quality  of  the  research  staff  and  of  the  management.  Close  behind  are  the 
quality  of  the  facilities  and  capital  equipment.  According  to  a  1991  report  to  the  Office  of  the 
Secretary  of  Defense,  an  effective  laboratory  will  exhibit  the  following  characteristics: 

.  A  clear  and  substantive  mission 
.  A  critical  mass  of  assigned  work 
.  A  highly  competent  and  motivated  work  force 
.  An  inspired,  empowered,  highly  qualified  leadership 
.  State-of-the-art  facilities  and  equipment 
.  An  effective  two-way  relationship  with  customers 
.  A  strong  foundation  in  research 

.  Management  authority  and  flexibility 

.  A  strong  linkage  to  universities,  industry,  and  other  government  laboratories.^^ 

These  statements  however  do  not  represent  the  whole  picture.  The  laboratory  program  must  be 
aligned  with  the  vision  and  mission  of  the  parent  organization;  in  this  case  the  U.S.  Army.  In  the 
previous  section  we  discussed  the  evaluation  of  the  portfolio  of  the  S&T  program  and  pointed 
out  that  it  is  aligned  with  the  needs  of  the  warfighter  through  continuing  discussions  with  the 
soldiers  and  their  representatives.  The  quality  of  the  S&T  product  depends  on  the  relevance  of 
the  work  to  the  needs  not  only  of  the  soldiers  but  also  of  the  initial  customers  of  the  work.  For 
example  the  Army  Materiel  Command’s  corporate  laboratory — ^ARL — ^works  on  problems 
defined  by  Army  TRADOC,  but  delivers  its  results  to  intermediaries  such  as  the  Research, 
Development  and  Engineering  Command  and  the  Program  Executive  Officers  and  Program 


"  Charles  E.  Adolph,  chairman,  Federal  Advisory  Commission  on  Consolidation  and  Conversion  of  Defense 
Research  and  Development  Laboratories,  Report  to  the  Secretary  of  Defense  (Washington,  DC:  Department  of 
Defense,  September  1991). 
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Managers  of  the  aequisition  eommunity.  They  ean  judge  the  immediate  impaet  on  their  work. 
The  ultimate  impaet,  also  a  measure  of  quality,  is  harder  to  evaluate  beeause  there  are  many 
hurdles  before  the  final  produet  or  proeess  is  fielded.  Even  when  fielded,  there  may  be  many 
years  of  experienee  needed  before  eonelusive  appraisals  ean  be  made.  Sinee  the  managers  of 
S&T  ean’t  wait  for  years  to  diseover  how  well  they  have  done,  they  need  some  way  of  evaluating 
the  work  in  the  short  term. 

The  evaluation  of  the  exeeution  of  the  program  eovers  three  phases:  the  planning  stage,  eurrent 
laboratory  work,  and  eompleted  work.  Planning  involves  evaluating  proposals  for  new  work  and 
assessing  them  for  relev anee;  the  availability  of  qualified  researeh  staff  ineluding  the  projeet 
leader;  the  mateh  to  faeilities  and  equipment  on  hand;  the  funding  neeessary  to  aequire  additional 
resourees  if  needed;  and  whether  or  not  the  ehallenge  of  the  problem  posed  is  sueh  that  it  ean  be 
solved  in  a  time  frame  that  is  praetieal  for  the  warfighter.  One  measure  of  quality  of  the  program 
is  the  extent  to  whieh  the  laboratory  or  laboratories  are  prepared  for  future  demands; 
preparedness  is  a  quality  that  requires  foresight  (and  teehnology  foreeasting)  and  flexibility.  The 
planning  phase  of  the  Army  S&T  program  is  eondueted  jointly  with  TRADOC  and  the  S&T’s 
eustomers  for  the  6.2  and  6.3  parts  of  the  program.  Reeently  TRADOC  has  beeome  involved  in 
the  6.1  program  in  terms  of  the  eoverage  of  the  areas  in  the  list  of  24  Army  priorities  for  S&T 
prepared  by  the  Deputy  Assistant  Seeretary  of  the  Army  for  Researeh  and  Development  in 
eoneert  with  TRADOC.  Mueh  of  the  planning  exereise  has  been  diseussed  in  Seetion  II  on  the 
portfolio. 

Peer  Review.  Evaluation  of  the  quality  of  eurrent  researeh  and  development  work  is  somewhat 
more  straightforward  than  evaluating  the  portfolio  in  that  peer  review  is  the  prime  evaluating 
tool.  Currently,  there  is  an  established  teehnique  for  peer  review  of  laboratories  whieh  ean  aid  in 
ensuring  quality  in  researeh  laboratories.  There  are  many  variations — different  laboratories 
may  require  different  approaehes.  If  the  peers  are  drawn  from  within  the  laboratory  or  its  parent, 
there  will  be  good  knowledge  of  the  teehnieal  area  and  the  ehallenges  being  addressed.  However, 
internal  judgments  may  be  biased  based  on  the  stakes  held  by  the  evaluators.  On  the  other  hand, 
external  evaluators  will  have  less  intimate  knowledge  of  the  speeifies  of  the  work  but  their 
evaluation  will  have  more  eredibility.  Eor  most  assessments,  eredibility  is  the  leading 
eharaeteristie  as  the  results  will  help  loeal  management  improve  their  work  or  even  ehange 
eourse.  Eor  government  laboratories,  peer  review  proeesses  will  help  in  justifying  budget 
doeuments.  Conelusions  from  various  studies  of  peer  review  of  laboratories  indieate  that  the  best 
way  to  do  this  is  as  follows: 

.  Contraet  the  evaluation  to  a  reeognized  and  respeeted  entity  sueh  as  the  National 
Researeh  Couneil  of  the  National  Aeademies. 

.  Have  the  eontraetor  manage  the  entire  proeess  from  pieking  evaluators,  planning 
agendas,  handling  meeting  logisties,  and  supervising  preparation  and  distribution  of 
reports. 

•  Appoint  panels  of  evaluators  that  eonsist  of  independent  subjeet  matter  experts  with 
knowledge  of  the  areas  of  interest. 


National  Research  Council,  Panel  for  Review  of  Best  Practices  in  Assessment  of  Research  and  Development 
Organizations,  Best  Practices  in  Assessment  of  Research  and  Development  Organizations  (Washington,  DC: 
National  Academies  Press,  2012). 
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.  Have  panels  evaluate  not  only  the  technical  work  presented  to  them,  but  also  the  quality 
of  the  technical  staff,  the  quality  of  the  facilities  and  equipment,  and  the  adequacy  of  the 
resources  provided. 

•  Seek  to  evaluate  the  culture  of  the  laboratory  and  the  role  of  upper  management  in  hiring, 
promoting,  and  motivating  and  retaining  the  staff 

•  Hold  out-briefing  sessions  with  the  staff,  prepare  written  reports,  and  indicate  the  panel’s 
impressions  of  the  follow-up  actions  by  the  laboratory  on  recommendations  from  the 
previous  report. 

Most  Army  laboratories  carry  out  some  form  of  peer  review,  but  not  all.  The  evaluations  run  the 
gamut  from  using  contractors  as  described  above  to  conducting  strictly  in-house  assessments.  It 
would  be  useful  for  all  Army  laboratories  to  consider  some  kind  of  independent  peer  review  of 
current  work.  One  model  is  the  review  of  ARL.  This  is  done  by  the  National  Research  Council 
and  follows  the  above  recommended  course  of  action.  ARL  reviews  began  in  the  1990s  and  have 
been  done  regularly  since.  The  evaluation  reports  are  published  by  the  National  Research 
Council  and  are  available  on  the  Web  site  of  the  National  Academies  Press. 

Completed  Work:  Metrics  and  Benchmarking.  Some  aspects  of  a  laboratory  can  be  assessed 
quantitatively  and  the  results  can  be  compared  to  those  at  other  laboratories,  preferably 
laboratories  that  are  widely  regarded  as  outstanding.  Metrics  may  include  a  variety  of  topics: 
staff  educational  level,  staff  age  distribution,  number  of  archival  publications  per  technical  staff 
member,  number  of  patents,  number  of  external  awards  and  other  recognitions,  elected 
memberships  to  honorary  societies  such  as  the  National  Academies,  and  so  on.  Such  metrics 
are  useful  as  management  tools,  but  caution  must  be  taken  in  using  them.  These  measures  do  not, 
of  themselves,  indicate  the  quality  of  the  entries.  As  an  example,  a  laboratory  can  have  a  number 
of  unused  patents  and  a  string  of  uninspiring  publications.  In  this  situation,  this  laboratory  would 
have  met  its  criteria  for  metrics,  but  be  performing  poorly.  In  comparison,  a  laboratory  could 
have  written  brief  papers  that  have  won  Nobel  prizes.'"^  The  quality  of  completed  work  as 
evidenced  in  publications,  patents,  and  various  prototypes  for  products  can  be  assessed  by 
sampling  done  by  outside  peers.  This  will  ensure  a  mix  of  metrics  and  qualitative  assessments, 
and  provide  a  far  greater  and  far  more  accurate  representation  of  effectiveness. 

Simultaneously,  the  Army  can  compare  itself  with  other  military  laboratories  to  give  a  more 
meaningful  assessment.  The  Naval  Research  Laboratory  would  be  a  prime  example  for  such  a 
comparison,  provided  the  assessment  used  similar  metrics.  Other  possibilities  for  comparative 
analysis  include  the  nature  of  the  portfolio,  such  as  percentages  of  basic  and  applied  research,  the 
amount  of  work  in  which  the  laboratory  is  a  world  leader  in  the  technology,  and  the  focus  on 
short-term  vs.  long-term  work. 

We  recently  have  provided  an  informal  report  to  the  Deputy  Assistant  Secretary  of  the  Army  for 
Research  and  Technology  in  which  we  listed  eight  factors  important  for  evaluation  of  quality  of 
the  S&T  program.  The  factors  include  quality  of  staff;  quality  managers;  peer  review;  relevance 
(inputs  from  customers  and  stakeholders);  technology  forecasting;  leveraging;  scientific  or 
professional  positions  and  the  Independent  Research  &  Development  (IR&D)  conducted  by 


Report  of  an  Expert  Panel  on  the  Future  of  Army  Laboratories,  Improving  Army  Basic  Research  (Arlington,  VA: 
RAND  2012),  50-51.  This  report  lists  39  metrics  for  evaluating  basic  research. 

'''  James  D.  Watson  and  Francis  Ft.  Crick,  “A  Structure  for  Deoxyribose  Nucleic  Acid,”  Nature,  vol.  171  (1953),  37- 


38. 
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DOD  contractors;  and  visibility  of  Army  S&T.  We  compared  our  list  against  the  list  of  metries 
taken  from  a  reeent  report  from  the  RAND  Arroyo  Center.'^  The  result  is  shown  in  Appendix  A. 
The  published  list  of  metries  amplifies  our  seven  fundamentals.  There  is  no  eontradietion  in  the 
two  lists. 

Impact  and  Relevance — Relations  with  Customers  and  Stakeholders.  A  quality  laboratory  will 
know  its  eustomers  and  be  elose  to  its  stakeholders  to  ensure  it  fulfills  their  expeetations  for 
relevance  and  timeliness.  The  laboratory  should  address  the  needs  in  the  priority  in  whieh  they 
are  presented  by  the  eustomers.  As  noted  earlier,  the  Army  laboratories  deliver  their  researeh 
findings  to  a  well-defined  set  of  eustomers:  namely,  the  aequisition  eommunity  and  the 
warfighter’s  representatives.  An  assessment  of  a  laboratory  should  include  discussions  held  with 
samples  of  these  eommunities.  One  way  to  determine  how  well  the  laboratory  is  serving  its 
eustomers  is  simply  to  invite  representatives  to  researeh  reviews  and  listen  to  their  eomments. 
Laboratories,  whieh  detail  a  portion  of  their  staff  to  aequisition  offiees  under  matrix 
management,  hold  regular  meetings  with  those  Program  Exeeutive  Offieers  or  Program 
Managers  to  review  progress.  One  possible  technique  is  to  poll  customers  via  a  ratings  sheet 
followed  by  serious  discussion  of  any  area  not  given  a  fully  sueeessful  grade. 

As  noted  in  the  introduction,  the  stakeholders  are  the  senior  leaders  of  the  Army,  DOD,  and  the 
taxpayers  as  represented  by  the  U.S.  Congress.  The  opinions  of  senior  Army  and  DOD  leaders 
are  usually  made  known  by  them  to  the  Offiee  of  the  Assistant  Seeretary  of  the  Army  for 
Aequisition,  Logisties,  and  Technology  (ASAALT).  To  enhance  relations  with  stakeholders, 
meetings  and  laboratory  tours  can  be  effective,  but  stakeholder  sehedules  make  visits  very  hard 
to  arrange.  The  ASAALT  and  sometimes  senior  assistants  to  the  prineipals  ean  be  persuaded  to 
make  visits  and  discuss  issues.  Congressional  hearings  are  too  formal  and  too  brief  for  effeetive 
communieation.  However,  eongressional  staffers  are  often  interested  in  the  details  and  enjoy 
making  laboratory  visits. 

Taking  Advantage  of  External  Expertise.  No  laboratory  ean  be  at  the  frontiers  of  knowledge  in 
every  subjeet  for  whieh  it  is  responsible.  Therefore,  it  behooves  the  laboratory  to  seek  out 
experts  to  learn  what  they  know,  to  eooperate  where  feasible,  and  to  enter  into  formal 
collaborations  where  justified.  There  is  no  plaee  for  insularity  in  today’s  technologieal  world, 
save  only  for  classified  work.  Assessors  of  a  laboratory’s  work  should  ask  the  staff  what  other 
experts  are  working  in  this  field.  Who  are  the  leaders?  How  does  this  laboratory  eompare?  If  the 
laboratory  is  not  among  the  world’s  elite  in  a  partieular  field — not  working  at  the  state  of  the 
art — then  it  should  be  in  active  eontact  with  those  who  are.  There  should  be  eooperative 
programs  and  perhaps  formal  eollaborations.  If  not,  then  the  laboratory  should  exit  the  field.  It  is 
a  measure  of  the  quality  of  the  laboratory  management  as  to  whether  it  is  addressing  these 
questions.  Use  of  eomparative  metries,  noted  above,  ean  be  helpful  in  benehmarking  the  work 
against  others. 

The  quality  of  the  preparedness  of  the  laboratory  for  future  demands  is  helped  by  a  regular 
program  of  S&T  foreeasting.  The  laboratory  should  be  looking  ahead  and  eomparing  what  it 
foreeasts  against  what  the  demands  on  it  are  likely  to  be.  Foreeasting  ean  be  done  in  different 
ways.  At  the  individual  or  small  group  level,  one  expeets  the  loeal  experts  to  have  a  good  sense 
of  where  their  diseiplines  are  heading.  At  higher  levels,  sueh  as  the  laboratory  or  its  parent. 


See  Report  of  an  Expert  Panel  on  the  Future  of  Army  Laboratories,  Improving  Army  Basic  Research  (Arlington, 
VA:  RAND,  2012). 


15 


forecasting  is  done  in  an  organized  fashion,  perhaps  on  contract.  Forecasts  of  individual  sciences 
or  technologies  should  be  combined  with  cross-disciplinary  and  multi-disciplinary  predictions. 
One  concept  is  convergence  forecasting  in  which  mixed  panels  of  experts  in  several  potentially 
related  disciplines  work  together  to  make  predictions  and  suggest  specific  research  programs.'^ 
Forecasting  should  be  a  recurring  activity  in  a  rolling  manner  such  that  any  given  area  is  looked 
at  every  5  to  10  years.  The  Deputy  Assistant  Secretary  of  the  Army  for  Research  and  Technology 
sponsored  a  very  detailed  forecast  of  all  areas  of  interest  20  years  ago.^’  Resuming  forecasting  is 
long  overdue.  Lack  of  a  forecasting  program  is  a  factor  in  judging  the  overall  quality  of  the  S&T 
effort. 

Quality  of  S&T  and  Its  Reputation.  The  reputation  of  an  S&T  program  governs,  to  a  large 
extent,  its  ability  to  secure  resources  for  existing  and  new  programs.  The  most  important 
audience  is  therefore  the  parent  of  the  program  and  other  stakeholders  with  a  financial  interest. 
Their  impression  of  the  laboratory  is  formed  in  part  by  the  technical  results,  but  also  by 
comments  from  outside  the  enterprise  from  customers  and  other  interested  people.  For  the 
laboratories,  in  particular,  external  opinion  has  been  a  problem.  Reputation  is  formed  by  the 
impressions  given  by  publications,  presentations,  external  awards  and  other  public  recognition, 
and  press  coverage.  Nothing  is  as  impressive  as  the  receipt  of  a  Nobel  Prize.  The  Naval  Research 
Laboratory  has  one;  the  National  Institute  of  Standards  and  Technology  has  four,  and  the 
National  Institutes  of  Health  have  received  five.  The  Army  has  none.  However,  over  the  years 
ARO  has  provided  support  for  research  that  produced  18  Nobel  prizes.  The  Army  does  not  have 
an  individual  member  elected  to  one  of  the  National  Academies.  In  the  recent  past  the  Army  has 
had  one.  These  omissions  are  at  least,  in  part,  because  the  effort  has  not  been  made  to  publicize 
the  technical  work  and  to  nominate  staff  members  for  prizes  and  awards.  The  Army  has  not  done 
the  public  relations  work  necessary  to  keep  the  laboratories’  work  in  the  public  eye.  Compare 
this  with  the  Naval  Research  Laboratory  and  its  reputation,  or  Bell  Laboratories,  which 
continually  educates  the  public  on  who  invented  the  transistor.  Some  laboratories  cooperate  with 
authors  writing  books  about  their  work  and  their  history.  Jon  Gertner’s  recent  best-selling  book 
on  the  history  of  the  Bell  Laboratories  has  received  rave  reviews  and  is  an  example  of  this 
cooperation.'^  Could  similar  public  relations  efforts  be  carried  out  for  Army  S&T?  In  fact,  it  was 
attempted  on  the  anniversary  of  the  introduction  of  the  first  general  purpose  electronic 
computer,  ENIAC  (Electronic  Numerical  Integrator  and  Computer).  Creation  of  the  ENIAC  was 
funded  by  the  Army’s  ballistics  program  at  Aberdeen  Proving  Ground  at  the  end  of  World  War 
IT 

Because  of  this  lack  of  aggressive  publicity,  the  reputation  of  Army  S&T  is  not  what  it  could  be. 
This  has  led  to  lukewarm  support  for  the  Army’s  S&T  budget,  lukewarm  even  on  the  part  of 
some  senior  Army  leaders.  At  best,  this  problem  produces  more  support  for  Army  contracts  and 


John  Lyons,  Richard  Chait,  and  Simone  Erchov,  ed.,  Improving  the  Army ’s  Next  Effort  in  Technology 
Forecasting,  Defense  &  Technology  Paper  73  (Washington,  DC:  Center  for  Technology  and  National  Security 
Policy,  September  2010);  Douglas  Kiserow  et.  ah.  Report  of  an  Army  Workshop  on  Convergence  Forecasting, 
Mechanochemical  Transduction,  Defense  &  Technology  Paper  95  (Washington,  DC:  Center  for  Technology  and 
National  Security  Policy,  July  2012). 

National  Research  Council,  STAR  21-Strategic  Technologies  for  the  Army  of  the  Twenty-First  Century 
(Washington,  DC:  National  Academies  Press,  1992). 

**  Jon  Gertner,  The  Idea  Factory:  Bell  Labs  and  the  Great  Age  of  American  Innovation  (New  York:  Penguin  Press, 
2012). 
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grants  than  for  in-house  work.  Sometimes  this  is  appropriate,  sometimes  it  is  not.  The  Army 
should  not  hide  its  S&T  quality.  Instead,  it  should  be  its  biggest  promoter  and  justifier. 

Quality  Summary.  This  diseussion  is  foeused  on  the  laboratory  on  the  assumption  that  the 
quality  of  the  S&T  program  as  a  whole  would  be,  to  a  large  extent,  the  sum  of  the  individual 
parts.  We  have  eonsidered  three  aspeets  of  laboratory  quality:  the  planning  phase,  the  eurrent 
work  in  the  execution  of  the  plans,  and  the  completed  technical  work.  Planning  includes  the 
alignment  of  the  plans  with  the  mission  of  the  laboratory  and  of  the  Army.  A  quality  plan 
includes  the  availability  of  quality  staff  (and  managers);  availability  of  needed  resources 
including  budget,  facilities,  and  equipment;  the  allocation  of  resources  to  a  balance  of  short-  and 
long-term  research;  and  a  regular  program  of  technology  forecasting  to  ensure  preparedness  of 
the  laboratory  for  future  developments  in  science  and  engineering  and  evolving  trends  in  threats 
to  the  nation. 

Some  form  of  peer  review  should  be  used  to  assess  current  laboratory  work.  We  believe  the  best 
approach  is  to  contract  for  panels  of  independent,  unbiased  subject  matter  experts.  These  panels 
should  be  asked  not  only  to  evaluate  the  current  research  and  development  as  presented  by  the 
staff  doing  the  work,  but  should  also  be  asked  to  review  the  status  of  the  facilities  and 
equipment.  In  the  course  of  talking  with  the  staff  and  walking  through  the  laboratories  and 
meeting  with  management,  the  panels  can  get  an  idea  of  its  culture  and  the  nature  of  the 
management  personnel.  Such  panel  reviews  should  be  done  for  each  sector  of  the  laboratory 
every  2  to  3  years.  The  degree  to  which  the  research  staff  interacts  with  and  is  cognizant  of  the 
work  of  others  in  the  field  is  an  important  factor  in  rating  the  laboratory.  Various  kinds  of 
cooperation  and  formal  collaborations  with  external  expertise  are  a  way  to  make  sure  that  the 
laboratory  is  at  the  state-of-the-art  level  and  is  pushing  the  frontiers.  Working  with  others  is  a 
good  way  to  be  prepared  for  future  developments. 

Completed  work  produces  impacts  on  the  Army.  Results  from  the  laboratory  reach  the  warfighter 
through  the  efforts  of  many  players  including  the  technical  work  in  the  offices  of  Program 
Executive  Officers  and  Program  Managers  in  the  acquisition  community.  Results  may  be 
transmitted  directly  to  industrial  firms  that  are  preparing  for  manufacturing  of  the  new  products 
or  using  the  new  processes  from  Army  research.  During  war  time,  some  results  obtained  in  the 
laboratory  are  transmitted  though  accelerated  procedures  directly  to  soldiers  on  the  battlefield. 
Laboratory  findings  are  summarized  in  publications  and  presentations  and  often  result  in  awards 
and  other  forms  of  recognition.  Satisfaction  with  the  laboratories’  work  in  terms  of  relevance, 
quality,  and  timeliness  should  be  tested  by  regular  sessions  with  customers  and  stakeholders. 

Comparisons  of  the  laboratory  with  other  laboratories  deemed  to  be  outstanding  can  be  made  via 
benchmarking.  In  this  approach  various  metrics  can  be  obtained  and  comparisons  made.  These 
would  include  external  recognition  as  well  as  items  that  can  be  counted,  such  as  numbers  of 
papers  and  patents,  percentages  of  Ph.D.s,  and  so  on.  Reputation  is  important  for  a  number  of 
reasons,  but  the  most  fundamental  is  the  impact  a  strong  reputation  will  have  on  the  ability  to 
secure  resources  from  year  to  year. 

Questions  to  be  asked  when  assessing  the  quality  of  a  research  and  development  organization 
should  include: 

•  Is  the  work  aligned  with  the  mission  and  priorities? 

•  What  is  the  level  of  education  of  the  staff?  (percentage  of  Ph.D.s) 

.  What  is  the  average  age  of  staff?  (retention  rate) 
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•  Do  the  managers  have  experienee  at  the  laboratory  beneh?  What  is  their  reputation?  How 
do  they  affeet  the  culture? 

.  What  is  the  nature  of  the  basic  or  exploratory  work?  How  much? 

•  How  closely  does  the  staff  work  with  experts  in  other  laboratories? 

.  What  is  the  nature  of  cooperative  work  with  outside  laboratories?  Does  the  laboratory 
have  formal  collaborations? 

.  What  is  the  quality  of  the  laboratory’s  products?  (number  of  transitions  of  results  to 
customers) 

.  For  each  technical  staff  member,  what  is  the  number  of  publications,  internal  reports, 
patents,  and  presentations? 

.  How  is  the  staff  recognized?  (awards,  elected  memberships) 

.  What  has  been  the  impact  of  the  program  on  warfighting?  (retrospective  studies) 

.  What  is  the  state  of  the  facilities? 

.  What  is  the  state  of  capital  equipment? 

.  What  are  the  results  of  benchmarking  against  peer  laboratories? 
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V.  SUMMARY  OF  REPORT  AND  RECOMMENDATIONS 

In  this  paper  we  focused  on  the  existing  portfolio  as  provided  by  the  ASAALT  Office.  Our 
object  was  to  provide  a  methodology  for  evaluating  only  this  existing  program.  However,  to 
make  a  complete  assessment  one  would  have  to  consider  what  topics  are  either  not  being 
addressed  or  not  being  addressed  adequately.  The  answers  would  assist  in  reorienting  the 
portfolio.  It  would  have  been  of  great  assistance  if  the  Army  had  a  current  version  of  the  Army 
Science  &  Technology  Master  Plan. 

We  classified  the  work  of  the  Army  S&T  program  in  three  aspects:  work  where  the  Army  is 
dominant  in  the  technology,  work  where  the  Army  shares  the  technical  lead  with  the  private 
sector,  and  areas  where  the  private  sector  leads.  The  priority  and  nature  of  the  research  program 
varies  with  the  category.  The  paper  discusses  some  details  of  evaluating  the  portfolio  and 
presents  a  list  of  questions,  the  answers  to  which  would  help.  The  evaluation  of  the  current 
research  work  in  executing  the  portfolio  covers  three  phases:  planning,  execution,  and  completed 
work.  This  paper  provides  another  list  of  questions  for  assessing  the  quality  of  the  current  work. 
We  discuss  the  quality  of  the  staff,  how  to  perform  peer  review,  consider  relations  with  outside 
experts,  and  ask  about  cooperation  and  collaborations  with  others.  The  paper  provides  an 
approach  to  reviewing  and  evaluating  the  quality  of  the  portfolio  itself  and  the  technical 
execution  of  the  programs  and  projects. 

We  recommend: 

•  The  Army  resume  publication  of  the  Army  Science  &  Technology  Master  Plan 
.  The  Army  evaluate  the  current  S&T  portfolio  along  the  lines  suggested  herein. 

.  The  review  should  identify  areas  that  are  underfunded  (or  overfunded)  and  suggest 
corrections. 

.  The  review  should  attempt  to  identify  areas  of  science  and  technology  not  included  in  the 
current  portfolio  but  which  should  be  included. 

•  The  evaluation  should  make  sure  that  every  program  is  directly  in  support  of  the  Army’s 
mission  and  vision. 
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VI.  APPENDIX 


Appendix  A  includes  eight  fundamentals  taken  from  a  draft  report  by  CTNSP'^  in  a  matrix  with 
a  series  of  metrics.  The  X’s  indicate  the  relevant  items  in  the  rows  that  match  the  elements  in  the 
column. 


S&T  Quality  Fundamentals 

Metrics 

Quality 

Staff 

Quality 

Mgrs 

Peer 

review 

Relevance 

Tech 

Fcstg. 

Leverage 

STs  & 
IR&D 

Visibility 

Distribution  by 

S&T  degrees, 
age  distribution, 
and  field  of 
research 

X 

X 

Fellowship  in 

professional 

societies 

X 

X 

Membership  in 
national 
academies  or 
similar  bodies 

X 

X 

Other 

professional 
recognition 
(prizes  and 
awards) 

X 

X 

Breakdown  by 
advanced 
degrees  and 
scientific  and 
engineering 
disciplines 

X 

Refresh  and 
turnover  rate  of 
staff 

X 

X 

Invited  and 

keynote 

speeches 

X 

X 

X 

Vision: 

commonality  of 
goals 

X 

X 

Funding  level 

X 

X 

X 

Consistency  of 
funding  levels 

X 

X 

Scientific 
reputation  of 
research 
managers 

X 

X 

Private  communication  to  the  Deputy  Assistant  Secretary  of  the  Army  for  Research  and  Technology,  June  6, 
2012. 
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S&T  Quality  Fundamentals 

Metrics 

Quality 

Staff 

Quality 

Mgrs 

Peer 

review 

Relevance 

Tech 

Fcstg. 

Leverage 

STs  & 
IR&D 

Visibility 

Number  of 
significant 
breakthroughs 
and  inventions  in 
the  past  10  years 

X 

X 

Recruitment  of 
high  quaiity 
researchers 

X 

X 

X 

Reievance  for 
the  enterprise 

X 

X 

Rainmaker: 
abiiity  to 
influence 
sponsors/funders 

X 

X 

X 

Seiection 
process: 
subfieids  to 
invest  and 
proposais. 

Support  for 
externai  review 
of  proposais  and 
research 

X 

X 

X 

X 

Success  in 
heiping 
technoiogy 
transfer 

X 

X 

X 

Portfoiio  view  of 

research 

activities 

(baiance 

between 

enhancement, 

extensions,  new 

chaiienges  in 

existing  areas, 

new  emerging 

areas) 

X 

X 

X 

Criteria  and 
assessment 
process  for 
researchers  and 
their  career 
advancement 

X 

X 

X 

Emphasis  on 
peer-reviewed 
competition  for 
basic  research 
funding 

X 

X 
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S&T  Quality  Fundamentals 

Metrics 

Quality 

Staff 

Quality 

Mgrs 

Peer 

review 

Relevance 

Tech 

Fcstg. 

Leverage 

STs  & 
IR&D 

Visibility 

Specification  of 
research 
process,  score 
cards 

X 

X 

X 

X 

Administrative 
burden  on 
researchers 

X 

Mentoring  of  new 
employees 

X 

X 

Operational 

management 

X 

X 

X 

X 

Intellectual 

stimulation 

X 

X 

X 

X 

X 

Unstructured 

activities 

X 

X 

X 

Interactions  with 
customers 

X 

X 

X 

X 

Support  for  new 
ideas 

X 

X 

X 

Lab  facilities; 

computing 

environment 

X 

X 

Publications 

X 

X 

X 

Citation  by 
others 

X 

X 

X 

Peer  recognition 

X 

X 

X 

Patents 

X 

X 

X 

X 

Citation  of 
patents  by  others 

X 

X 

X 

X 

Tech  transfer  to 
field 

X 

X 

X 

X 

Long-term 
impact  on  users 
and  funders 

X 

X 
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